{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading and Writing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We have already talked about built-in Python types, but there are more types that we did not speak about. One of these is the ``file()`` object which can be used to read or write files."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try and get the contents of the file into IPython. We start off by creating a file object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/data.txt', 'r')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The ``open`` function is taking the [data/data.txt](data/data.txt) file, opening it, and returning an object (which we call ``f``) that can then be used to access the data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that ``f`` is not the data in the file, it is what is called a *file handle*, which points to the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now, simply type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The ``read()`` function basically just read the whole file and put the contents inside a string."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try this again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's happened? We read the file, and the file 'pointer' is now sitting at the end of the file, and there is nothing left to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To close the file handle, you can do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's now try and do something more useful, and capture the contents of the file in a string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/data.txt', 'r')\n",
    "data = f.read()\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now ``data`` should contain a string with the contents of the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "But what we'd really like to do is read the file line by line. There are several ways to do this, the simplest of which is to use a ``for`` loop in the following way:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data/data.txt', 'r')\n",
    "for line in f:\n",
    "    print(repr(line))\n",
    "\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note that we are using ``repr()`` to show any invisible characters (this will be useful in a minute). Also note that we are now looping over a file rather than a list, and this automatically reads in the next line at each iteration. Each line is being returned as a string. Notice the ``\\n`` at the end of each line - this is a line return character, which indicates the end of a line."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now we're reading in a file line by line, what would be nice would be to get some values out of it.  Let's examine the last line in detail. If we just type ``line`` we should see the last line that was printed in the loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can first get rid of the ``\\n`` character with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line = line.strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Next, we can use what we learned about strings and lists to do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns = line.split()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Finally, let's say we care about the object name (the 2MASS column), and the J band magnitude (the Jmag) column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name = columns[2]\n",
    "jmag = columns[3]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jmag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Note that ``jmag`` is a string, but if we want a floating point number, we can instead do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jmag = float(columns[3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "jmag"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One last piece of information we need about files is how we can read a single line. This is done using:\n",
    "\n",
    "    line = f.readline()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can put all this together to write a little script to read the data from the file and display the columns we care about to the screen! Here is is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()\n",
    "# Open file\n",
    "f = open('data/data.txt', 'r')\n",
    "\n",
    "# Read and ignore header lines\n",
    "header1 = f.readline()\n",
    "header2 = f.readline()\n",
    "header3 = f.readline()\n",
    "\n",
    "# Loop over lines and extract variables of interest\n",
    "for line in f:\n",
    "    line = line.strip()\n",
    "    columns = line.split()\n",
    "    name = columns[2]\n",
    "    jmag = float(columns[3])\n",
    "    print(name, jmag)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python also provides a way to read each line in a text file as a separate string (returns a list). In the following example, we read the lines in the same command line, immediately after opening the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()\n",
    "# Open the file and read each line separately\n",
    "mydata = open('data/data.txt','r').readlines()\n",
    "print (mydata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(mydata[4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a copy of the above code to read in the file. Modify this code so as to create a dictionary which gives ``Jmag`` for a given ``2MASS`` name, i.e.\n",
    "\n",
    "    >>> jmag['00424455+4116103']\n",
    "    10.773\n",
    "    \n",
    "Then loop over the items in the dictionary and print out for each the source name and the ``Jmag`` value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EDIT THE CODE BELOW\n",
    "\n",
    "# Open file\n",
    "f = open('data/data.txt', 'r')\n",
    "\n",
    "# Read and ignore header lines\n",
    "header1 = f.readline()\n",
    "header2 = f.readline()\n",
    "header3 = f.readline()\n",
    "\n",
    "d = {}\n",
    "# Loop over lines and extract variables of interest\n",
    "for line in f:\n",
    "    line = line.strip()\n",
    "    columns = line.split()\n",
    "    name = columns[2]\n",
    "    jmag = float(columns[3])\n",
    "    d[name] = jmag\n",
    "\n",
    "f.close()\n",
    "\n",
    "for key in sorted(d):\n",
    "    print (key, d[key])\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus:** can you figure out a way to make sure that you loop over the source names in alphabetical order?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To open a file for writing, use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f = open('data_new.txt', 'w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Then simply use ``f.write()`` to write any content to the file, for example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.write(\"Hello, World!\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you want to write multiple lines, you can either give a list of strings to the ``writelines()`` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.writelines(['roof\\n', 'tile\\n', 'roof\\n'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "or you can write them as a single string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.write('roof\\ntile\\nroof\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Once you have finished writing data to a file, you need to close it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(this also applies to reading files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The with-statement"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen above, files must not just be opened but should be properly closed afterwards to make sure they are actually written before using them somewhere else. Sometimes writes to files get cached by Python to minimize actual writing to disk, which is comparably slow. Closing a file ensures that these changes are actually written.\n",
    "\n",
    "To avoid forgetting to close a file there is the with-statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('data/data_new.txt', 'w') as f:\n",
    "    f.write('spam\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This opens the specified file and holds the file-object within ``f``, as well as closing the file when the with-codeblock ends. Afterwards, the file is properly closed and not available anymore."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What's behind the ``with`` is called a “context manager”.  This is a far more general concept, which tends to be useful whenever you need to maintain “external invariants” – which is jargon for “cleaning up after yourself”.  For instance, you can use context managers to reliably remove temporary files, log out from services that have some session management, stop background jobs started, and even reset things within your program. If you come back here later, you'll understand the next piece of language: Check out the contextlib module."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Work with the file data/autofahrt.txt\n",
    "\n",
    "Continuing from the example in the 'Reading files' section, read in columns one (=time $t$) and one of columns two to four (=acceleration $\\ddot x(t)$, $\\ddot y(t)$, $\\ddot z(t)$). You choose which one.\n",
    "\n",
    "Then, write out the two columns to a new file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# EDIT THE CODE BELOW\n",
    "\n",
    "# Open file\n",
    "f = open('data/autofahrt.txt', 'r')\n",
    "\n",
    "# Read and ignore header lines\n",
    "header1 = f.readline()\n",
    "header2 = f.readline()\n",
    "\n",
    "# Loop over lines and extract variables of interest\n",
    "for line in f:\n",
    "    line = line.strip()\n",
    "    columns = line.split()\n",
    "    time = columns[0]\n",
    "    a = float(columns[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The above shows you how you can read and write any data file. Of course, there are also functions that exist to help you read in data in certain formats (for example ``numpy`` contains a function ``numpy.loadtxt`` to read in arrays from files) but the key is that with the above, you can read in any file."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}